The present invention relates to an accident and crime preventive technique for measuring the height of vehicles at the entrances of parking lots, measuring the height of people going on board vehicles, and monitoring shops and roads, also relates to a method and apparatus for detecting and measuring moving objects for analysis of images taken by cameras, and further concerns a recording medium for recording a program for detecting and measuring a moving object.
Investigations are being made to impose height limits for cars and human beings at the entrances of parking lots or at the doors of wagons in the amusement parks, etc. A criminal's height is judged by visual measurement from ex post facto images recorded by time plus video recorders for surveillance in convenience stores and so on. Further, In addition to the admission regulations in terms of height, there are vehicle admission regulations for narrow road widths.
In the parking lots, poles and sign boards are provided, which limit the height of entering vehicles. At the entrance to a parking lot, when the car contacts the height-limiting pole, for example, a warning is given to the driver, or information is supplied to the attendant to make a decision to refuse admission or to designate another parking area. At the entrance to vehicles in an amusement park, there is sometimes what may be called a height check board having a hole in the shape of a human being to limit the height of people going on board, and by comparing with a height comparison-purpose panel, a warning is issued to a person concerned or information is supplied for the attendant to refuse admission for security reasons. In convenience stores, there is a color-classified height measuring tape at the exit, the customers are monitored at all times by a monitor camera and a time plus video recorder. After an incident has occurred, the height of a criminal is determined by visual measurement from the recorded images. On narrow-width roads, there are signs prohibiting the entrance of vehicles or large-sized vehicles. The comparison-purpose panels, stickers, and road signs are intended to preclude incidents or gather information by measuring the height or the width of objects, which are going to enter, in specified places (hereafter referred to as moving objects). Under present circumstances, however, analysis of image records has to rely on visual inspection or decision by a human being for technical reasons. Therefore, there has been demand for automated measuring process using a computer, and various methods have been proposed.
As a method for detecting a moving object, there has been proposed “Apparatus and Method for Detecting and Extracting a Moving Body” in JP-A-8-221577. This method realizes detection and extraction of the feature of a moving object against a complicated background and also realizes a reduction in image processing time. Referring to FIG. 2, this method will be described in the following.
In FIG. 2, frame images F1 (241) to F7 (245) are the frame images of the images taken at time T1 (221) to T5 (225). A segment S (231) in the frame image in FIG. 2 is designated as that area of the input image to which attention is directed in monitoring, and this area of interest is hereafter referred to as a slit. Reference numerals 201 through 205 in FIG. 2 denote slices of images at the slit (hereafter referred to as slit images) and a background image taken at time T1 (221) through time T5 (225). In this example, the image of the area of interest in monitoring when no moving object is not being captured by the camera is set as the background image at the start of processing.
In this method, the following steps are carried out on each frame image. (1) Slit images and the corresponding portions of the background images in a specific frame are extracted, (2) an extent of variation in the image structure of the slit image and the background image is obtained, (3) when the amounts of structure variation in those images are viewed as time series, if the image structure variation appears in the form of waves, it is decided that there is a moving object, and (4) when the image structure variation is flat without showing any change, it is decided that the background with no moving object superposed is shown.
The step (3) will be described using a sequence of frame images in FIG. 2. When an object moves across the slit as in this example, the image structure variation appears in a wavy pattern as shown in the image structure change graph (211) in FIG. 2.
The image structure variation here is a measure by which to indicate to what extent the structure of an object, such as the position or shape of an object in the image, has changed. The image structure variation is a feature quantity not affected by illuminance, and differs from an image variation representing a variation in pixels in the image. The method of calculating the image structure variation will be described in detail later on.
Before an object comes into the slit (time T1 (221)), the image at the slit and the corresponding portion of the background image are almost the same (201), the image structure variation is little. Then, when the object starts to cross the slit (time T2 (222)), the slit image differs from the background image (202), and image structure variation becomes large. Finally, when the object has passed through the slit (time T3 (223)), the image structure variation returns to a small value. As an object passes through the slit, the image structure variation changes in a wavy pattern. Therefore, to find a moving object, it is only necessary to observe the image structure variation as a time series and find a change in an upward convex form. In this example, to recognize this upward convex change, the moment when the image structure variation exceeds a threshold value a (213) and the moment when it falls below the threshold value a (213) are used.
The step (4) mentioned above will be described using the sequential frame images shown in FIG. 2. If a baggage (252) is left to stand about the slit as in this example (time T4 (224)), the image structure variation is large at first, but because the baggage (252) is at a standstill, the image structure variation remains unchanged at a high level (from time T4 (224) till time T5 (225)). In this method, when the value of the image structure variation has been small for more than a fixed time period as in this example, the background is changed automatically by using the slit image at this point in time as the background.
Lastly, the method for calculating the image structure variation in the step (2) will be described with reference to FIGS. 13A, 13B, 13C, 14A, 14B, 14C and 15. FIGS. 13A, 13B and 13C show the effects that the changes in illuminance have on the slit images. Let us consider a slit 1301, in which the illuminance of each pixel is expressed by P1 . . . PN as show in FIG. 13A. If we draw a graph in which the vertical axis indicates the illuminance of the pixels and the horizontal axis indicates the position of the pixels, we have a graph of a slit image 1305 in FIG. 13B. Supposing the slit image 1301 as a whole becomes dark due to a sudden change in illuminance, such as by shadow, the graph 1301 undergoes a uniform decrease from 1305 to 1307 while retaining the relative illuminance levels among the pixels P1 through PN as shown in FIG. 13C. This change in illuminance can be illustrated as shown in FIGS. 14A, 14B and 14C by expressing the slit image by vector notation.
As shown in FIG. 14A, the slit image can be regarded as a vector V (1401) made up of unit vectors, which represent the illuminance of the pixels. If base vectors of the pixels P1 through PN are designated as b1, b2, b3 . . . bn, the vector (1401) can be expressed as a point in a n-dimensional vector space as in FIG. 14B. Then, let us suppose that a shadow fell over the slit image and the illuminance changed suddenly, with a result that the slit vector V (1401) changed to a slit vector V′ (1403) as shown in FIG. 14C. At this time, from observations of FIG. 3, the changed slit vector V′ (1403) can be considered to be almost on the same segment as the vector V (1401) and also considered to be a scalar multiple of the vector V (1401). The slit vector V′ (1403), formed as a result of a change in illuminance in the original slit vector V 1401, remains in almost the same direction even though the slit vector V′ (1403) is located at coordinates greatly different from the coordinates of the vector V (1401) in the vector space. In contrast, a slit vector, which has undergone a structural change, is considered to be different not only in coordinate values but also in direction. Therefore, to differentiate a change in illuminance from a change in structure in the slit image 1301, it is only necessary to know whether there is a change in the direction of the vector.
FIG. 15 shows an ordinary slit vector V (1401) and the slit vector V′ (1403) affected by a change in illuminance, and their projected shadows on a unit sphere. As shown in FIG. 15, the distance PQ between a projected vector P (1501) on the unit sphere as a shadow of the vector V (1401) and a projected vector Q (1503) on the unit sphere as a shadow of the vector V′ (1403) is much closer to each other than the original distance VV′. What occurred in the two slit images, which differentiates one from the other, that is, whether a difference occurred due to a change in illuminance or a structural change, can be decided by finding whether or not the distance between the projected vectors on the unit sphere is very small. The normalized distance between the vectors is here-after referred to as a normalized distance. By using the normalized distance, the degree in which a structural change occurred in an objet in an image can be obtained. In this method, the image structure variation is used in detection of a moving object in the step (3) or in detection and automatic changing of the background image as mentioned in the description of the step (4). Thus, at the time of changeover of the lighting condition from day to night or night to day or under the condition that an object is placed at the background position, which has conventionally made monitoring difficult, it becomes possible to correctly decide whether there is a moving object, whether there is a new background or whether the change is merely by a shadow.
The outline of “Apparatus and Method for Detecting and Extracting a Moving Object” has been described. In this method, a segment can be used as an area of interest to be monitored; therefore compared with the conventional method, which uses the whole screen image as an area of interest, it becomes possible to greatly reduce time for calculation of the image structure variation. Further, in this method, by monitoring changes on the time base of the image structure variation, it is possible to find timing for changing the background and therefore even in places where the background is likely to be changed, as in taking pictures outdoors, a monitor process can be applied. In this method, because image variations as difference amounts of the pixels are not used but degrees of structural change in the object in the image are used, the background and the moving object can be distinguished from each other even when the lighting condition is changed or a structural change occurred in the background image.
Further, “Apparatus and Method for Detecting and Extracting a Moving Object by Using Combined Information” have been disclosed in JP-A-11-134506. In this method, there have been provided a plurality of areas of interest used in the moving object detecting apparatus, and the position and the size of a moving object are decided from a combination of some pieces of moving object detection information captured by the lattice-type monitoring areas. This method will be described with reference to FIG. 3.
In FIG. 3, the vertical position and the horizontal position of a moving object 341 in an input TV screen image 300 are detected by using a group of slits (311 through 315 and 321 through 324) arranged in a lattice form. This group of slits are formed by arranging a group of vertical slits, arranged like vertical lines, such as slit V1 (311), slit V2 (312), slit V3 (313), slit V4 (314) and slit (315), so as to be perpendicular to a group of horizontal slits, arranged like horizontal lines, such as slit H1 (321), slit H2 (322), slit H3 (323) and slit H4 (324). The vertical slits are arranged in parallel spaced apart by a distance of Lw (332). Similarly, the horizontal slits are arranged in parallel spaced apart by a height of Lh (331).
In FIG. 2, while the image structure variation is higher than the threshold value a (213), specifically, an interval from time T2 (222) till time T3 (223) and an interval on and after time T4 (224), moving object detecting means continues issuing a moving object detection event (for example, if there are 30 picture frames per second, those frames are checked one frame after another to see if there is a moving object. While the image structure variation is higher than the threshold value, a signal indicating that a moving object is detected continues being issued.)
As a slit identifier of moving object detection information, character strings indicating names of the slits, such as “V1”, “V2”, “H1” and “H4”, are set to identify each of the slits shown in FIG. 3.
Description will now be made of a method for deciding the size of a moving object by using the slits mentioned above. In FIG. 3, in the case of a moving object 341, the vertical slits V2 (312) and V3 (313) and the horizontal slits H2 (322) and H3 (323) are detected. Among the vertical slits, the slit V1 (311) at the left of the slit V2 (312) and the slit V4 (314) at the right of the slit V3 (313) are not detected. Among the horizontal slits, the slit H1 (321) above the slit H2 (322) and the slit H4 (324) below the slit H3 (323) are not detected. The width of the moving object is known to be greater than the width Lw (332) according to the detected vertical slits V2 (312) and V3 (313) and less than twice the width Lw (332) according to the vertical slits V1 (311) and V4 (314) not detected. The height of the moving object is known to be greater than the height Lh (331) according to the detected horizontal slits H2 (322) and H4 (324) and less than twice the height Lh (331) according to the horizontal slits H1 (321) and H4 (324) not detected.
Description has been made of the outline of “Apparatus and Method for Detecting and Extracting a Moving Body by Using Combined Information” by which to decide the size of the moving object. In this method, the height, width and position of a moving object is determined by using a combination of multiple pieces of information obtained from input images captured by a lattice-type moving object detecting means. With this method, because decisions are made on the time base, it is possible to keep track of a moving object on the input TV screen image.